Improve assertions error messages with structured format#7444
Improve assertions error messages with structured format#7444Evangelink wants to merge 57 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request modernizes assertion error messages by introducing a structured, multi-line format that improves readability and developer experience. The changes replace old-style concatenated messages with formatted parameter displays using raw string literals.
Changes:
- Introduced structured error message formatting with parameter names, expressions, and values on separate lines
- Added helper methods for value formatting, expression truncation, and redundancy detection
- Updated all assertion methods to use the new formatting approach
- Removed obsolete resource strings and added new simplified ones
- Updated all test expectations to match the new message format
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Assert.cs | Core formatting infrastructure: FormatValue, FormatParameter, truncation logic |
| Assert.ThrowsException.cs | Updated to new structured format for exception type mismatches |
| Assert.StartsWith/EndsWith/Matches.cs | String assertion methods using new format |
| Assert.IsTrue/IsFalse.cs | Boolean assertion methods with condition parameter display |
| Assert.IsNull/IsNotNull.cs | Null checking assertions with value display |
| Assert.IsInstanceOfType/IsExactInstanceOfType.cs | Type checking with structured type comparison |
| Assert.IComparable.cs | Comparison assertions (greater/less than, positive/negative) |
| Assert.Count/Contains.cs | Collection assertions with expression parameters |
| Assert.AreSame.cs | Reference equality with hash code display for disambiguation |
| FrameworkMessages.resx | Resource string simplification and new message keys |
| xlf files | Localization files marked with untranslated entries |
| Test files | Updated expectations for all assertion error messages |
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsInstanceOfTypeTests.cs
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs
Outdated
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsTrueTests.cs
Show resolved
Hide resolved
a55c762 to
49018c6
Compare
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs
Outdated
Show resolved
Hide resolved
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs
Outdated
Show resolved
Hide resolved
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf
Show resolved
Hide resolved
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf
Show resolved
Hide resolved
49018c6 to
63e8257
Compare
|
looks great |
User message now appears right after the call-site line, before the assertion explanation and parameter details. The 'User message:' prefix is removed - position alone distinguishes it.
- Converge all assertion messages to 'Expected [subject] to [verb phrase].' style - Add ellipsis (...) to callsite for overloads with omitted params (delta, culture) - Show delta, ignoreCase, culture as structured parameters - Rework CollectionAssert to use AppendUserMessage + FormatAlignedParameters - Rework StringAssert to use new structured format - Rework Assert.That to strip lambda wrapper and generate expression-aware messages (comparisons, bool members, negation, StartsWith/Contains/All/Any) - Update all test assertions to verify full messages
- Fix #1: Replace hardcoded 'Expected all elements...' with AllMatchPredicateFailNew resource - Fix #2: Remove 11 dead resource strings (ContainsFail, StartsWithFail, EndsWithFail, IsMatchFail, IsNotMatchFail, DoesNotEndWithFail, DoesNotStartWithFail, AssertThatFailedFormat, AssertThatMessageFormat, AssertThatDetailsPrefix, CollectionEqualReason) - Fix #3: Replace bare catch with catch (Exception) in TryEvaluateFormatted - Fix #5: Type-check StartsWith/EndsWith/Contains to only match string methods - Fix #6: Add explicit ellipsis sentinel handling in FormatCallSite - Fix #7: Fix grammar 'Both collection contain same elements' -> 'Both collections contain the same elements'
- Fix IDE0028/IDE0306: Use collection expression [.. collection] instead of new List<T>(collection) - Fix IDE0046: Simplify if-return to conditional expression in BuildMethodCallMessage - Fix IDE0051: Remove unused private methods (IsExpressionRedundant, TryExtractStringLiteralContent, IsExpressionMoreSpecificNumericLiteral, FormatParameterWithValue, FormatParameterWithExpressionCheck) - Fix SA1028/IDE0055: Remove trailing whitespace in IsInstanceOfTypeTests.cs - Fix CS8137/CS8179: Replace ValueTuple params with StringPair struct to support net462 (no System.ValueTuple dependency) - Fix AreSame.cs: Restore missing method signatures and closing braces
| catch (Exception) | ||
| { | ||
| // If enumeration fails, report what we've collected so far | ||
| truncated = elements.Count > 0; | ||
| } | ||
|
|
||
| private static string BuildUserMessageForNotExpectedPrefixExpressionAndValueExpression(string? format, string notExpectedPrefixExpression, string valueExpression) | ||
| => BuildUserMessageForTwoExpressions(format, notExpectedPrefixExpression, "notExpectedPrefix", valueExpression, "value"); | ||
| int totalCount = knownCount ?? enumeratedCount; | ||
| int displayedCount = elements.Count; | ||
|
|
||
| private static string BuildUserMessageForPatternExpressionAndValueExpression(string? format, string patternExpression, string valueExpression) | ||
| => BuildUserMessageForTwoExpressions(format, patternExpression, "pattern", valueExpression, "value"); | ||
| string elementList = string.Join(", ", elements); | ||
| if (truncated) | ||
| { | ||
| int remaining = totalCount - displayedCount; | ||
| string remainingText = knownCount is null | ||
| ? $"{remaining}+" | ||
| : $"{remaining}"; | ||
| elementList += $", ... {remainingText} more"; | ||
| } |
There was a problem hiding this comment.
When FormatCollectionPreview catches an exception during enumeration, it forces truncated = elements.Count > 0, but enumeratedCount may equal displayedCount (e.g., exception thrown after the last successfully added element). In that case remaining becomes 0 and the preview can end up with ... 0+ more, which is misleading. Consider tracking whether enumeration was actually cut short (e.g., a separate failedEnumeration flag) and only append the ... N more suffix when remaining > 0, or use a non-numeric suffix for the “unknown remainder due to enumeration failure” case.
- OutputTests: Update expected AreEqual message to new callsite format - TrxReportTests: Update expected AreEqual message with aligned spacing - RetryTests: Split Assert.Fail message check (no longer 'Assert.Fail failed.') - GenericTestMethodTests: Update regex to match new Assert.Fail format (callsite on separate line)
- Fix FormatCollectionPreview: Use failedEnumeration flag instead of forcing truncated=true, avoiding misleading '... 0+ more' suffix on exception - Fix CollectionAssert.AreEquivalent null check: Use dedicated 'Expected collections to be equivalent.' message with expected/actual nullness instead of generic AreEqualFailNew - Fix FormatValue culture: Use Convert.ToString with InvariantCulture for primitive types to ensure consistent output across locales
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 55 out of 55 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (1)
src/TestFramework/TestFramework/Assertions/Assert.cs:120
FormatValuetreats anyIEnumerableas a collection preview and will enumerate it to build the message. This can consume one-shot / lazy / side-effecting enumerables (and may be inconsistent withFormatCollectionParameter, which avoids enumerating non-ICollectionsequences). Consider only previewingICollection(or materialized snapshots) and falling back to type name for non-ICollectionenumerables to avoid changing runtime behavior during message formatting.
// For collections, show a preview with element values
if (value is IEnumerable enumerable)
{
return FormatCollectionPreview(enumerable);
}
|
|
||
| private static string BuildUserMessageForTwoExpressions(string? format, string callerArgExpression1, string parameterName1, string callerArgExpression2, string parameterName2) | ||
| internal static string ReplaceNulls(object? input) | ||
| => input?.ToString() ?? string.Empty; |
There was a problem hiding this comment.
ReplaceNulls now returns an empty string when input is null, which will produce blank values in assertion messages that still use ReplaceNulls (e.g., mismatched element reporting in CollectionAssert). Consider returning a consistent null token (e.g., "null" or FrameworkMessages.Common_NullInMessages) so nulls are visible in formatted output.
This issue also appears on line 116 of the same file.
| => input?.ToString() ?? string.Empty; | |
| => input?.ToString() ?? FrameworkMessages.Common_NullInMessages; |
| string msg = "Expected collections to be equivalent."; | ||
| msg += Assert.FormatAlignedParameters( | ||
| new Assert.StringPair("expected", expected is null ? "null" : "not null"), | ||
| new Assert.StringPair("actual", actual is null ? "null" : "not null")); | ||
| msg = Assert.AppendUserMessage(msg, message); | ||
| Assert.ThrowAssertFailed("CollectionAssert.AreEquivalent", msg); |
There was a problem hiding this comment.
The nullability-mismatch failure message is currently hardcoded ("Expected collections to be equivalent." and "null/not null"), while most other assertion messages are coming from FrameworkMessages.resx. To keep assertion output localizable and consistent, consider moving these strings into resources (and using the same null token as the rest of the formatter).
See below for a potential fix:
if ((expected is null) != (actual is null))
{
string msg = "Expected collections to be equivalent.";
msg += Assert.FormatAlignedParameters(
new Assert.StringPair("expected", expected is null ? Assert.ReplaceNulls(null) : "not null"),
new Assert.StringPair("actual", actual is null ? Assert.ReplaceNulls(null) : "not null"));
| ? " Objects are equal." | ||
| : " Objects are not equal."; |
There was a problem hiding this comment.
The equality diagnostic hint strings ("Objects are equal." / "Objects are not equal.") are hardcoded. Since assertion messages are generally localized via FrameworkMessages.resx, consider moving these hint fragments into resources (or otherwise ensuring they follow the same localization strategy).
| ? " Objects are equal." | |
| : " Objects are not equal."; | |
| ? FrameworkMessages.AreSameObjectsEqualHint | |
| : FrameworkMessages.AreSameObjectsNotEqualHint; |
| // Negation: !flag, !condition | ||
| case UnaryExpression unary when body.NodeType == ExpressionType.Not | ||
| && unary.Operand is not MethodCallExpression: | ||
| string innerName = GetCleanMemberName(unary.Operand); | ||
| return $"Expected {innerName} to be false."; | ||
|
|
||
| // Method calls: text.StartsWith("x"), list.Contains(item), etc. | ||
| case MethodCallExpression methodCall: | ||
| return BuildMethodCallMessage(methodCall); | ||
|
|
||
| // Bool member access: user.IsActive, flag | ||
| case MemberExpression: | ||
| string memberName = GetCleanMemberName(body); | ||
| return $"Expected {memberName} to be true."; | ||
|
|
||
| default: | ||
| return FrameworkMessages.IsTrueFailNew; | ||
| } | ||
| } | ||
|
|
||
| private static string BuildComparisonMessage(BinaryExpression binary) | ||
| { | ||
| string leftValue = TryEvaluateFormatted(binary.Left); | ||
| string rightValue = TryEvaluateFormatted(binary.Right); | ||
|
|
||
| return binary.NodeType switch | ||
| { | ||
| ExpressionType.Equal => $"Expected {leftValue} to equal {rightValue}.", | ||
| ExpressionType.NotEqual => $"Expected {leftValue} to not equal {rightValue}.", | ||
| ExpressionType.GreaterThan => $"Expected {leftValue} to be greater than {rightValue}.", | ||
| ExpressionType.GreaterThanOrEqual => $"Expected {leftValue} to be greater than or equal to {rightValue}.", | ||
| ExpressionType.LessThan => $"Expected {leftValue} to be less than {rightValue}.", | ||
| ExpressionType.LessThanOrEqual => $"Expected {leftValue} to be less than or equal to {rightValue}.", | ||
| _ => FrameworkMessages.IsTrueFailNew, |
There was a problem hiding this comment.
Assert.That now builds several user-facing failure sentences via hardcoded interpolated strings (e.g., "Expected {x} to be false/true" and the comparison messages). If this framework’s assertion output is intended to be localizable (as suggested by the extensive FrameworkMessages.resx usage), these new strings should also be moved into resources for consistency and localization coverage.
- Fix ReplaceNulls: Return 'null' instead of empty string for null input - Move AreSame equality hints to resx (AreSameObjectsAreEqualHint/AreSameObjectsAreNotEqualHint) - Move AreEquivalent null mismatch message to resx (AreEquivalentNullMismatchFailNew) - Move Assert.That comparison/bool messages to resx (AssertThatEqualFailNew, AssertThatNotEqualFailNew, AssertThatGreaterThanFailNew, etc.) - All user-facing strings now go through FrameworkMessages for localization
- ContainsSingle predicate tests: Add wildcard for new collection preview line - Items interpolated string tests: Remove collection lines, add wildcards - IsInRange DateTime: Use wildcards for InvariantCulture date format - AreEqual culture: Add ... to expected callsite for culture overload - Assert.That char: Update expected values for InvariantCulture char formatting
| private static string TryEvaluateFormatted(Expression expr) | ||
| { | ||
| try | ||
| { | ||
| object? value = Expression.Lambda(expr).Compile().DynamicInvoke(); | ||
| return FormatValue(value); | ||
| } | ||
| catch (Exception) | ||
| { | ||
| return GetCleanMemberName(expr); |
There was a problem hiding this comment.
TryEvaluateFormatted compiles and executes sub-expressions (Expression.Lambda(expr).Compile().DynamicInvoke()), which can re-run user code after the initial condition.Compile()() evaluation. This can cause side effects (methods invoked twice), performance regressions, or different results if expressions are non-deterministic. Consider limiting evaluation to safe expression types (constants/member access) or avoid executing method calls when building the failure message.
| // For collections, show a preview with element values | ||
| if (value is IEnumerable enumerable) | ||
| { | ||
| return FormatCollectionPreview(enumerable); |
There was a problem hiding this comment.
FormatValue<T>(T? value, int maxLength = 256) ignores the maxLength argument for IEnumerable values because it calls FormatCollectionPreview(enumerable) without passing maxLength. This makes FormatValue(..., maxLength: X) behave inconsistently depending on the runtime type. Pass maxLength through to FormatCollectionPreview (and/or document that collections are always formatted with a fixed limit).
| return FormatCollectionPreview(enumerable); | |
| return FormatCollectionPreview(enumerable, maxLength); |
| [DoesNotReturn] | ||
| private static void ThrowAssertDoesNotContainPredicateFailed(string userMessage) | ||
| private static void ThrowAssertIsInRangeFailed<T>(T value, T minValue, T maxValue, string? userMessage, string minValueExpression, string maxValueExpression, string valueExpression) | ||
| { | ||
| string finalMessage = string.Format( | ||
| CultureInfo.CurrentCulture, | ||
| FrameworkMessages.DoesNotContainPredicateFailMsg, | ||
| userMessage); | ||
| ThrowAssertFailed("Assert.DoesNotContain", finalMessage); | ||
| string callSite = FormatCallSite("Assert.IsInRange", new StringPair(nameof(value), valueExpression)); | ||
| string message = string.Format(CultureInfo.CurrentCulture, FrameworkMessages.IsInRangeFailNew, FormatValue(value), FormatValue(minValue), FormatValue(maxValue)); | ||
| message += FormatAlignedParameters( | ||
| new StringPair("range", $"[{FormatValue(minValue)}, {FormatValue(maxValue)}]"), | ||
| new StringPair(nameof(value), FormatValue(value))); | ||
| message = AppendUserMessage(message, userMessage); |
There was a problem hiding this comment.
ThrowAssertIsInRangeFailed receives minValueExpression and maxValueExpression but never uses them. This makes the signature misleading and increases the chance of future callers assuming those expressions affect the output. Either remove these parameters or include them in the call-site/details (e.g., show the full call site / range expression).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (4)
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:309
- The non-generic
Contains(object?, IEnumerable)eagerly materializes non-ICollectioninputs viaToList()before searching. This changes passing behavior (a match near the start no longer short-circuits) and can cause large/infinite/side-effecting sequences to be fully enumerated unnecessarily. Consider scanning the enumerable directly and only materializing/buffering elements when an assertion failure requires a stable snapshot for message formatting.
// Materialize non-ICollection enumerables to prevent multiple enumeration.
ICollection snapshot = collection as ICollection ?? collection.Cast<object?>().ToList();
foreach (object? item in snapshot)
{
if (object.Equals(item, expected))
{
return;
}
}
ThrowAssertContainsItemFailed(message, expectedExpression, collectionExpression, snapshot);
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:391
Contains<T>(Func<T,bool>, IEnumerable<T>)snapshots non-ICollectionenumerables ([.. collection]) before callingAny. That preventsAnyfrom short-circuiting and can introduce significant overhead/side effects on passing assertions. Consider enumerating once and stopping at the first match, buffering only what’s needed for the failure message.
public static void Contains<T>(Func<T, bool> predicate, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(predicate))] string predicateExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
{
ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
if (!snapshot.Any(predicate))
{
ThrowAssertContainsPredicateFailed(message, predicateExpression, collectionExpression, snapshot);
}
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:524
DoesNotContain<T>snapshots non-ICollectionenumerables ([.. collection]) before checkingContains. This prevents early-exit on failures (whennotExpectedappears early) and can fully enumerate expensive/side-effecting sequences unnecessarily. Consider iterating and failing immediately on the first match; only buffer/materialize when needed for formatting the failure message.
public static void DoesNotContain<T>(T notExpected, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(notExpected))] string notExpectedExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
{
ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
if (snapshot.Contains(notExpected))
{
ThrowAssertDoesNotContainItemFailed(message, notExpectedExpression, collectionExpression, snapshot);
}
src/TestFramework/TestFramework/Assertions/Assert.Contains.cs:633
DoesNotContain<T>(Func<T,bool>, IEnumerable<T>)snapshots non-ICollectionenumerables ([.. collection]) before callingAny. This removesAny’s ability to short-circuit and can fully enumerate sequences even when the predicate matches early (the failing case). Consider scanning the enumerable directly and stopping at the first match, buffering elements only when producing the failure message requires it.
public static void DoesNotContain<T>(Func<T, bool> predicate, IEnumerable<T> collection, string? message = "", [CallerArgumentExpression(nameof(predicate))] string predicateExpression = "", [CallerArgumentExpression(nameof(collection))] string collectionExpression = "")
{
ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection];
if (snapshot.Any(predicate))
{
ThrowAssertDoesNotContainPredicateFailed(message, predicateExpression, collectionExpression, snapshot);
}
| // Materialize non-ICollection enumerables to prevent multiple enumeration | ||
| // that could yield different results or fail on second pass. | ||
| ICollection<T> snapshot = collection as ICollection<T> ?? [.. collection]; | ||
| if (!snapshot.Contains(expected)) | ||
| { | ||
| string userMessage = BuildUserMessageForExpectedExpressionAndCollectionExpression(message, expectedExpression, collectionExpression); | ||
| ThrowAssertContainsItemFailed(userMessage); | ||
| ThrowAssertContainsItemFailed(message, expectedExpression, collectionExpression, snapshot); | ||
| } |
There was a problem hiding this comment.
Contains<T> materializes non-ICollection enumerables ([.. collection]) before checking membership. This forces full enumeration even when the expected item is found early, which is a behavior and performance regression (and can surface iterator side-effects/exceptions that previously wouldn't occur on passing assertions). Consider iterating once and returning as soon as the item is found, only buffering elements when you need to build the failure message (or only materializing on the failure path).
This issue also appears in the following locations of the same file:
- line 298
- line 385
- line 518
- line 627
See below for a potential fix:
if (collection is ICollection<T> collectionOfT)
{
if (!collectionOfT.Contains(expected))
{
ThrowAssertContainsItemFailed(message, expectedExpression, collectionExpression, collectionOfT);
}
return;
}
List<T> snapshot = [];
foreach (T item in collection)
{
snapshot.Add(item);
if (EqualityComparer<T>.Default.Equals(item, expected))
{
return;
}
}
ThrowAssertContainsItemFailed(message, expectedExpression, collectionExpression, snapshot);
| Assert.ThrowAssertFailed("CollectionAssert.AllItemsAreUnique", finalMessage); | ||
| string msg = FrameworkMessages.AllItemsAreUniqueFailMsg; | ||
| msg += Assert.FormatAlignedParameters( | ||
| new Assert.StringPair("duplicate", Assert.ReplaceNulls(current))); |
There was a problem hiding this comment.
In AllItemsAreUnique, the duplicate value is formatted via Assert.ReplaceNulls(current) which does not escape newlines or apply truncation/quoting. Since this PR introduces Assert.FormatValue specifically to keep failure messages single-line per parameter and consistent, consider using Assert.FormatValue(current) (or equivalent) for the duplicate parameter value.
| new Assert.StringPair("duplicate", Assert.ReplaceNulls(current))); | |
| new Assert.StringPair("duplicate", Assert.FormatValue(current))); |
| { | ||
| Assert.ThrowAssertFailed("CollectionAssert.IsSubsetOf", $"{returnedSubsetValueMessage} {userMessage}"); | ||
| } | ||
| string missingElements = string.Join(", ", isSubsetValue.Item2.Select(item => Convert.ToString(item, CultureInfo.InvariantCulture))); |
There was a problem hiding this comment.
IsSubsetOf builds missingElements via Convert.ToString(item, ...) and string.Join. If an element is null, Convert.ToString returns null and string.Join will render it as an empty string, losing information in the failure message. Consider mapping nulls explicitly (e.g., item is null ? "null" : ...) or using the new Assert.FormatValue/Assert.ReplaceNulls helpers for each element before joining.
| string missingElements = string.Join(", ", isSubsetValue.Item2.Select(item => Convert.ToString(item, CultureInfo.InvariantCulture))); | |
| string missingElements = string.Join(", ", isSubsetValue.Item2.Select(item => Assert.FormatValue(item))); |
- Add safety comment to TryEvaluateFormatted about re-evaluation side effects - Fix FormatValue: Pass maxLength to FormatCollectionPreview for consistency - Fix ThrowAssertIsInRangeFailed: Remove unused minValueExpression/maxValueExpression params - Fix Contains/DoesNotContain: Avoid eager materialization when ICollection is available - Fix AllItemsAreUnique: Use FormatValue instead of ReplaceNulls for proper escaping - Fix IsSubsetOf: Use FormatValue instead of Convert.ToString for null-safe formatting - Update IsSubsetOf tests for quoted string values
Fix #7421